Papers with data labeling
Crowdsourcing Natural Language Data at Scale: A Hands-On Tutorial (2021.naacl-tutorials)
Copied to clipboard
| Challenge: | a tutorial on crowdsourcing for efficient data annotation will introduce crowdsourcing and provide an overview of the technology. |
| Approach: | This tutorial will introduce users to efficient data annotation via crowdsourcing marketplaces. |
| Outcome: | This tutorial will introduce users to the use of crowdsourcing for data annotation. |
Explaining Why: How Instructions and User Interfaces Impact Annotator Rationales When Labeling Text Data (2022.naacl-main)
Copied to clipboard
Jamar Sullivan Jr., Will Brackenbury, Andrew McNutt, Kevin Bryson, Kwam Byll, Yuxin Chen, Michael Littman, Chenhao Tan, Blase Ur
| Challenge: | In the context of data labeling, researchers are interested in having humans select rationales . |
| Approach: | They conducted an online user study to understand how humans select rationales . they found that participants were near unanimous in their data labels . |
| Outcome: | The results show that participants selected 12% of input tokens as rationales, but fewer if unable to drag over multiple tokens at once. |
Toward Recognizing More Entity Types in NER: An Efficient Implementation using Only Entity Lexicons (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing named entity recognition systems require large scale labeled data to perform, while annotation of NER data is laborious and time-consuming. |
| Approach: | They propose to adjust an existing named entity recognition system to recognize entity types not defined in the system. |
| Outcome: | The proposed method can be quickly adjusted to a named entity recognition system. |
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models (2023.acl-long)
Copied to clipboard
| Challenge: | Existing methods to measure stereotypes in large language models rely on manual templates or natural sentences that contain stereotypes. |
| Approach: | They propose a prompt-based method to measure stereotypes in large language models . they use natural language descriptions of the target demographic group alongside unmarked defaults . |
| Outcome: | The proposed method detects that portrayals contain higher rates of racial stereotypes than human-written portrayals. |
From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learning (2025.findings-emnlp)
Copied to clipboard
| Challenge: | In-context learning (ICL) enables large language models to perform novel tasks without parameter updates by conditioning on a few input-output examples. |
| Approach: | They propose a cost-efficient two-stage pipeline that reduces reliance on LLMs for data labeling. |
| Outcome: | The proposed pipeline reduces reliance on LLMs for data labeling . it leverages readily available cross-task examples to prompt an LLM and pseudo-label a small set of target task instances. |
Unsupervised Expressive Rules Provide Explainability and Assist Human Experts Grasping New Domains (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Understanding unexplored data is a slow process, and there is no labeled data at hand. |
| Approach: | They propose to use unsupervised methods to reveal rules which cluster unexplored corpus by its prominent categories to help domain experts understand their texts. |
| Outcome: | The proposed rules can be bootstrapped to identify target categories and deepen understanding of the data. |
DA-Pred: Performance Prediction for Text Summarization under Domain-Shift and Instruct-Tuning (2025.emnlp-main)
Copied to clipboard
| Challenge: | Large Language Models (LLMs) often don’t perform as expected under Domain Shift or after Instruct-tuning. |
| Approach: | They propose a method that uses the known performance in high-resource domains and fine-tuning settings to predict performance in low-resourced domains or base models. |
| Outcome: | The proposed method can help researchers decide if resources should be allocated for data labeling and LLM Instruct-tuning. |
Sampling Bias in Deep Active Classification: An Empirical Study (D19-1)
Copied to clipboard
| Challenge: | Existing studies on active learning identify sampling bias in large datasets . cost and time needed for labeling and model training are bottlenecks preventing new and/or better models from being trained . |
| Approach: | They propose to use active learning to identify representative data samples for training . they propose to create tiny datasets that can be used for cheap training if needed . |
| Outcome: | The proposed model outperforms the state-of-the-art on active text classification using small representative datasets with active learning. |
Distillation with Explanations from Large Language Models (2024.lrec-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) like ChatGPT and GPT-4 have made remarkable progress in various NLP tasks while also providing explanations alongside their answers. |
| Approach: | They propose to use Large language models (LLMs) to generate more accurate answers and corresponding free-text explanations by combining ground truth labels and answers-explanations generated by LLMs. |
| Outcome: | The proposed method achieves improved predictive performance and generates explanations that exhibit greater alignment with the model’s task outputs. |
Reasoning Makes Good Annotators : An Automatic Task-specific Rules Distilling Framework for Low-resource Relation Extraction (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to extract knowledge from unlabeled data generate noise labels. |
| Approach: | They propose an automatic task-specific rules distilling framework to generate a logic rule from unlabeled data. |
| Outcome: | The proposed framework could power the labeling ability by discovering reliable model-labeled data. |
Efficient Data Labeling by Hierarchical Crowdsourcing with Large Language Models (2025.coling-main)
Copied to clipboard
| Challenge: | Large language models (LLMs) have been gaining attention for their impressive performance in in-context dialogues. |
| Approach: | They propose a hierarchical framework that leverages multiple LLMs for efficient data labeling under budget constraints. |
| Outcome: | The proposed framework outperforms human labelers and GPT-4 in terms of accuracy and efficiency. |
ICL-Bandit: Relevance Labeling in Advertisement Recommendation Systems via LLM (2025.findings-emnlp)
Copied to clipboard
Lu Wang, Chiming Duan, Pu Zhao, Fangkai Yang, Yong Shi, Xuefeng Luo, Bingjing Xu, Weiwei Deng, Qingwei Lin, Dongmei Zhang
| Challenge: | In-context learning (ICL) is a common practice to enhance LLM performance on domain-specific tasks. |
| Approach: | They propose a method that leverages large language models to enhance query-ad relevance labeling . they identify and provide superior demonstrations for ICL, thereby improving labeling performance . |
| Outcome: | The proposed method improves query-ad relevance labeling performance by providing demonstrations. |
Zero-Shot Spoken Language Understanding via Large Language Models: A Preliminary Study (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have shown promising results in zero-shot settings, which motivates us to explore prompt-based methods. |
| Approach: | They propose a two-stage framework which transforms the SLU task into a question-answering problem by directly prompting LLMs. |
| Outcome: | The proposed framework can be built by directly prompting LLMs to understand user needs without training data. |